Notes on Big-n Problems

where α is a step size. If the gradient of f is Lipschitz-continuous with constant L, then using the step size α = 1/L guarantees that f(x) < f(x) (provided ∇f(x) 6= 0), and if f is convex that the distance to the optimal value f(x∗) as a function of k is O(1/k). Nesterov [1983] proposes a simple modification, referred to as the accelerated gradient method, that improves this to O(1/k), and under certain assumptions this is the fastest possible convergence rate. In this work we review methods that take advantage of the problem structure to yield alternative (or complementary) methods for improving the convergence rate in cases where n is extremely large and the functions fi are ‘similar’. For example, in the degenerate case where each fi is identical the incremental gradient method discussed in the next section has a convergence rate of O(1/kn) in terms of k and n using a step size of n/L. That is, the error rate is decreased n times faster than the basic gradient method. Despite being a slower convergence rate in terms of k, when n is large and only a finite number of iterations will be performed, this may yield a more practical method than an optimal O(1/k) methods. Although in general we don’t expect to achieve an n-fold speed-up since the functions fi will not be identical, we might still expect to see computational gains in cases where the functions fi are sufficiently similar.